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ABSTRACT. There is a well-known sequence of constants c n de- 
scribing the growth of supercritical Galton- Watson processes Z n . 
With "lower deviation probabilities" we refer to P(Z n = k n ) with 
k n = o(c n ) as n increases. We give a detailed picture of the 
asymptotic behavior of such lower deviation probabilities. This 
complements and corrects results known from the literature con- 
cerning special cases. Knowledge on lower deviation probabilities is 
needed to describe large deviations of the ratio Z n +x/Z n . The lat- 
ter are important in statistical inference to estimate the offspring 
mean. For our proofs, we adapt the well-known Cramer method 
for proving large deviations of sums of independent variables to 
our needs. 
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1. Introduction and statement of results 

1.1. On the growth of supercritical processes. Let Z = (Z n ) n >o denote a 
Galton- Watson process with offspring generating function 

/(«) = E^ si > o< s < i, (i) 

j>0 

which is required to be non-degenerate, that is, pj < 1, j > 0. Suppose that Z 
is supercritical, i.e. /'(I) =: m 6 (l,oo). For simplicity, the initial state Zo > 
1 is always assumed to be deterministic, and, if not noted otherwise (as by an 
application of the Markov property), we set Z = 1. 

It is well-known (see, e.g., Asmussen and Hering (1983) P §3.5]) that 

there are c n > such that a.s. c^ x Z n — ► some non-degenerate W. (2) 

nfoo 

In this sense, the sequence of constants c n describes the order of growth of Z. But, 
P(W = 0) = q, with q G [0, 1) the smallest root of f(s) = s, that is, the extinction 
probability of Z. On the other hand, W restricted to (0, oo) has a (strictly) positive 
continuous density function denoted by w. Therefore the following global limit 
theorem holds: 

/>oo 

lim P(Z n > xcn) = / w{t) dt, x > 0. (3) 

n|oo J x 

The normalizing sequence (c„)„>o can be chosen to have the following additional 
properties: 

c = 1 and c„ < c„ +1 < mc n , n > 0, (4a) 
c n = m n L(m n ) with L slowly varying at infinity, (4b) 
lim L(x) exists; it is positive if and only if EZilogZi < oo. (4c) 

x]oc 

Because of (@p,c), we may (and subsequently shall) take 

Cn := m n if EZilogZi < oo. (5) 

1.2. Asymptotic local behavior of Z, purpose. A local limit theorem related 
to is due to Dubuc and Seneta (1976) ^Uj, see also P §3.7]. To state it we need 
the following definition. 

Definition 1 (Type (d, //)). We say the offspring generating function / is of type 
(d, ju), if d > 1 is the greatest common divisor of the set { j — I : j ^ £, pjpe > 0}, 
and fi > is the minimal j for which pj > 0. O 

Here is the announced local limit theorem. Suppose / is of type (d,fj,). Take 
x > 0, and consider integers k n > 1 such that k n /c n — * x as n f oo. Then, for 
each j > 1, 

lim (cn P{Z„ = k n \Z =j} - d l{jfe n =j /i »( mo dd)} w j( x )) = °j ( 6 ) 
j 

where Wj :— ^ (^) q?~ l w* e . 

£=1 

In particular, in our standard case Zq = 1 and if additionally k n = (i n (modd), 
then 

P(Z n = k n ) ~ dc^ 1 w(k n /c n ) as n | oo (7) 
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(with the usual meaning of the symbol <~ as the ratio converges to 1). 

Statement © [and especially Q] can be considered as describing the local be- 
havior of supercritical Galton- Watson processes in the region of normal deviations 
(from the growth of the c n ; 'deviations' are meant here in a multiplicative sense, 
related to the multiplicative nature of branching). But what about P(Z n = k n ) 
when k n /c n — » or oo ? In these cases we speak of lower and upper (local) devia- 
tion probabilities, respectively. 

Lower deviations of Z n are closely related to large deviations of Z n +i/Z n (see 
Ney and Vidyashankar (2004) |151 Section 2.3]). The latter are important in sta- 
tistical inference for supercritical Galton- Watson processes, since Z n+ i/Z n is the 
well-known Lotka-Nagaev estimator of the offspring mean. 

The main purpose of the present paper is to study lower deviation probabilities 
in their own and to provide a detailed picture (see Theorems 0] and below). As a 
starting point we discuss a relevant claim in 15 concerning an important special 
case (see Sections ll.4l and lL"5"l below) . Applications of our results for large deviations 
of Z n+ i/Z n and also to subcritical Galton- Watson processes are postponed to a 
future paper. 

Here is the program for the remaining introduction. After introducing a basic 
dichotomy, we review in Sections 11.41 and 11.51 what is known on lower deviations 
from the literature, before we state our results in Sections II .61 and ll .71 

1.3. A dichotomy for supercritical processes. Recalling that / denotes the 
offspring generating function, q the extinction probability, and m the mean, 

set 7 := f'(q), and define a by 7 = m~ a . (8) 

Note that 7 £ [0, 1) and a € (0, 00]. We introduce the following notion, reflecting 
a crucial dichotomy for supercritical Galton-Watson processes. 

Definition 2 (Schroder and Bottcher case). For our supercritical offspring 
law we distinguish between the Schroder and the Bottcher case, in dependence on 
whether p + p\ > or = 0. O 

Obviously, / is of Schroder type if and only if 7 > 0, if and only if a < 00. 

Next we want to collect a few basic facts from the literature concerning that 
dichotomy. Clearly, / can be considered as a function on D, where D denotes the 
closed unit disc in the complex plane. As usual, denote by /„ the n th iterate of /. 
We start with the Schroder case. Here it is well-known (see, e.g., ^ Lemma 3.7.2 
and Corollary 3.7.3]) that 

00 

X>M zeD. (9) 

3=0 

Moreover, the convergence is uniform on each compact subsets of the interior D° of 
D. Furthermore, the function S restricted to the reals is the unique solution of the 
so-called Schroder functional equation (see, e.g., Kuczma (1968) |13l Theorem 6.1, 
p.137]), 

S(f(s)) = 7 S(a), 0<s<l, (10) 

satisfying 

S{q) = and limS'(s) = 1. (11) 



S n (z) := In[Z) n q — » some S(z) =: 
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As a consequence of J§J, 

lim 7-" P(Z„ = fc) = v k , k> 1. (12) 

n joo 

Consequently, in the Schroder case, these extreme (k is fixed) lower deviation prob- 
abilities P(Z n = k) are positive and decay to with order 7™. On the other hand, 
the characteristics a G (0, 00) describes the behavior of the limiting quantities 
w(x) and P(W < x) as x J. 0. In fact, according to Biggins and Bingham (1993) 
[S], there is a continuous, positive multiplicatively periodic function V such that 

x^wix) = V{x) + o(l) as i|0. (13) 

Dubuc (1971) (7| has shown that the function V can be replaced by a constant 
Vq > if and only if 

S(tp(h)) = K h- a , h>0, (14) 

for some constant K > 0, where ip = (p\y denotes the Laplace function of W, 

y w {h) := Ee- hw , h>0. (15) 

We mention that condition (|14fl is certainly fulfilled if Z is embeddable (see ^ 
p. 96]) into a continuous-time Galton- Watson process (as in the case of a geometric 
offspring law, see Example |3] below) . 

Now we turn to the Bottcher case. Here fj, > 2 (recall Definition^. Clearly, 
opposed to H12[). extreme lower deviation probabilities disappear, even P(Z n < 
fi n ) = for all n > 1. Evidently, 

P(Z n = M «) = P(Z n _! = ^- 1 )p ( f" ) . (16) 

Hence, 

n—l n -, 

P(Z n = fx n ) = Hp[^ = exp[^— -log J. (17) 

Next, P(Z n = M n + 1) = P(Z„_! = ^- 1 ) A i n - 1 p H . lj ^ n ~ 1 - 1 . Thus, from 

P(Z„ = M " + 1) = p^+i M"- 1 P(Z„ = M n )- (18) 

For simplification, consider for the moment the special case p^+j > 0, j > 0. Then, 
as in the previous representation, for fixed k > and some positive constants Ck , 

P(Z n = n n + k) ~ c k n nk P(Z n = fi n ) as nfoo. (19) 

Consequently, in contrast to (fT5|) in the Schroder case, here the lower positive 
deviation probabilities P(Z n = \i n + k) do not have a uniform order of decay. But 
by CS1), 

lT n \ogP{Z n = // l + k) — ► \ogp^ k > 0. (20) 

That is, on a logarithmic scale, we gain again a uniform order, namely — fj, n . 
Turning back to the general Bottcher case, 

lim (f n (s)) {ti ~ n) =: B(s), Q < s < 1, (21) 

n|oo 

exists, is continuous, positive, and satisfies the Bottcher functional equation 

B(/(«)) = B"(a), < s < 1, (22) 
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with boundary conditions 

B(0) = and B(l) = 1 (23) 

(see, e.g., Kuczma (1968) [TJ Theorem 6.9, p.145]). 
Recalling that fi > 2, define [3 £ (0, 1) by 

/i = mP. (24) 

According to Theorem 3], there exists a positive and multiplicatively periodic 
function V* such that 

-\ogV{W<x) = x-W-VV*[x)+o(x-W-V) asx|0. (25) 

If additionally \ogipw(h) ~ —nhP as h | oo for some constant n > 0, then by 
Bingham (1988) ® formula (4)], 

- log P(W < x) ~ /T 1 (1-/3) (/s/^/a-fl x -/s/(i-fl as ijO. (26) 

1.4. Lower deviation probabilities in the literature. What else is known in 
the literature on lower deviation probabilities of Z ? In the Schroder case ( < 
a < oo), Athreya and Ney (1970) J2J proved that in case of mash d = 1 and 
EZ 2 < oo, for every e £ (0,7y), where 

77 := m Q /( 3+Q > > 1, (27) 

there exists a positive constant C £ such that for all k > 1, 

m"P(Z„ = fc)-™(fc/m")| < C £ £^ + (77 - e)- n . (28) 

The estimate (|28|) allows to get some information on lower deviation probabilities. 
Indeed, in the general Schroder case, from ljT3)l. 

u;(ac) x a;" -1 as x | (29) 

(meaning that there are positive constants C\ and C2 such that C\x a ~ x < w(x) < 
C2 x" -1 , < a; < 1). Together with QgJ this implies 

cm (a — l)n 



P(Z n = k n ) = m- n w(k n /m n ) 



1 + 



as n | 00. (30) 



^™ A£ -1 (»7-e) B > 

We want to show that in important special cases the O-expression is actually an 
o(l). Recalling the definition (|27(l of 77, one easily verifies that m an /k^rf 1 — > 
(as n | 00) if and only if fc„/m n ( 2+Q )/( 3+ ") — ► 00. Concerning the second O- 
term, if additionally a < 1, then to( q_1 )"//c" _1 < 1 provided that k n < to™. 
Hence, here to^" -1 )"/ (A:" -1 (77 — e) n ) converges to zero if r\ — e > 1. On the other 
hand, if a > 1 and k n /m, n ( 2+a ^( 3+a ) — > 00 (which we needed for the first term), 
then m^ a -^ n / (k"' 1 (77 - e) n ) -> provided that additionally e < to q /( 3 +") - 
TO ( Q - 1 )/( 3 + Q ). Altogether, under the assumptions in [5], 

P(Z n = k n ) = m~ n w(k n /m n ) (l + o(l)) as n f 00 (31) 

provided that both fc„ < to" and fc„/m n ( 2+Q )/( 3+Q ' — > 00. 

In [2] it is also mentioned that according to an unpublished manuscript of S. Kar- 
lin, in the Schroder case, for each embeddable processes Z of finite second moment, 

lim — — j- P(Z n = k n ) exists in (0, oo), provided that k n = 0(777"). (32) 

"Too kn 
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In the present situation, as we remarked after (|13|) . w(x) ~ Vb x a 1 as x J, with 
Vb > 0. Hence, from (1321) . for some constant C > 0, 

P(Z„ = fc n ) ~ Cm- n w(k n /m n ) as n f oo, (33) 

which is compatible with (I31|l . 

Intuitively, the asymptotic behavior of lower deviation probabilities should be 
more related to characteristics as a and (3 than to the tail of the offspring distribu- 
tion. Thus one can expect that it is possible to describe lower deviation probabilities 
successfully without the second moment assumption used in Actually, in |15l 
Theorem 1] one finds the following claim. 

Suppose po — and EZilogZi < oo. Then there exist positive constants 
Ci < C2 such that for k n — > 00 with k n = 0(m n ) as n j 00, 

„ . ,. . c P{Z n = k n ) P\Z n = k n ) , /„ .n 

Ci < hmmf < hmsup < G 2 , (34) 

cxd -^n 

where 

r p?^ 1 if " < 1, 
A n := J if a = 1, (35) 

[ m _n if 1 < a < 00, 

and := [n + 1 — logfc„/logm]. Furthermore, if k n — vn n ~ ln for natural num- 
bers i n = 0(n) as n | 00, then 

lim An 1 P(Z n = k n ) =: Cum exists in (0, 00). (36) 

rofoo 

1.5. Contradictions. Let us test that claim by an example which allows explicit 
calculations. 

Example 3 (Geometric offspring law). Consider the offspring generating func- 
tion 

oc 

f(s) = 8 = Vm-^l-m-^-V, 0< S <1, (37) 

to — (m — lis * — ' 

3=1 

(with mean m > 1). Obviously, here g = 0, 7 = to -1 , hence a = 1. For the n th 
iterate one easily gets 

00 

/4 s ) - 7^ TV" = 'ym- n (l-m- n ) j - 1 s j . (38) 

V ' TO" - (TO™ - 1)S 

3=1 

Thus, 

P(Z n = fc) = m" n (1 - m,- n ) k - 1 < to"™, (39) 

for all n,k > 1. On the other hand, since pi = to" 1 , by claim l|34(l there is a 
constant C > such that for the considered fc„ , 

P{Z n = k n ) > C9 n m- n (40) 

for n large enough. If, for example, k n — ra n / 2 then 9 n — > 00, and (|40|l contradicts 
(|39|l . Consequently, the left-hand part of claim l|34|l cannot be true in the case 
a= 1. O 
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Next we compare the claim with our discussion in the previous section on lower 
deviation probabilities based on |2J. In fact, under the assumptions in [2], if ad- 
ditionally k n = o(m n ) but fc„/m" (2+a)/(3+Q) -> oo as n | oo, then by ^TJ and 



P(^n = fcn) 



£• \Q-1 

-n / "'n 



(41) 



Thus, in the case 1 < a < oo we get P(Z n = k n ) = o(m~ n ) which contradicts the 
positivity of C\[ m in claim l[36[l . hence of C\ in claim l[34l) . 

Here is one more consideration. According to claim l|34[) . under 1 < a < oo, 

P(Z n = k) > Cm- n (42) 

for all k G [m En , m ( - 1_e ^ Tl ], £ G (0,1/2), and all n large enough. Here and later, 
C refers to a generic positive constant which might change its value from place to 
place. Hence, 



EZ- 1 > J2 k- 1 V{Z n = k) 



(43) 



> CmT n fc_1 = C(l-2e)nmT n (l + o(T)) as n | oo. 



k— m E 



But by Ney and Vidyashankar (2003) ^] Theorem 1], EZ„ 1 is asymptotically 
equivalent to m~™ (in the case 1 < a < oo), getting one more contradiction. 

Looking into details of the proof of [151 Theorem 1] , the following formulas are 
claimed to be true: 



27T Cli m = 

5>«7**(1), 



(44) 



3>1 



7r /m 
"7r/m 

— 7r/j7 



[sty(u))-Sty(-u)) 



du, 



a < 1, 



a = 1, 



7r/r 

V'(w) dw, 



//(^(«))+//(^(-«)) 



7r/r 



du, 1 < a < oo, 



a = oo, 



with S from @ and where ip = ipw denotes the characteristic function of W, 

ip w {u) := Ee mW , ueR. (45) 

Recall that Ci; m > according to the claim. Now, if a < 1, the positiveness of 
Cum is obvious from this formula, since the density function w is positive. But the 
point is that the claim Cn m > is not true in all other cases. 

In fact, consider first the case 1 < a < oo. It is well-known that ip solves the 
equation 

i/)(mu) = f(tp{u)), ugK, (46) 
(e.g. ^ formula (6.1)]). Iterating, we obtain 

$(m l u) = fe(i>(u)), uGR, £>l. (47) 
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Thus, 



/ 7r/m 

Therefore. 



/<(iK«))+//(iK-«)) 



du = TO 



0) + du - (48) 



E< 

< 



/i(^(tt))+/<(^(-«)) 



7r/m 



7r/m 



du 



du, 



(49) 



which is finite, since in the Schroder case (see, for example, [3], p. 83, Lemma 1), 



|^(u)| < c\u\~ a , 



u e 



Hence, 

E 5 



£>0 



ir/m 



£(V(tt)) + £(iK-u)) 



du = 



(50) 



+ / )V(u)du, (51) 

7r/m 



-7r/m /• oo 



and, consequently, by ll4~4l . 



1 

2^ 



■0(u) du 



in the present a € (l,oo) case. Inverting (|45() gives 

e - %ux ip(u) du = 27rw(x), x > 0. 



(52) 



(53) 



But by (|13|l there is a (positive) constant C such that w(x) < Cs" -1 , < x < 1. 
Hence, u>(0) = 0, and lt5^1l implies Cu m = 0. 

In the case a = oo, the proof of Lemma 5 in 15 is incorrect. In fact, the 
statement (82) there is wrong. But we can start from (79) there (setting r)(r, s) = 
1) to define 



77 fm 



7i(V».+r-i(«))d«, r,s>l, 0<j<r, (54) 



where in this section by an abuse of notation, 

i] H (u) := ft{e lu ' mt ) = Ee mZe/m \ i > 0, uE 



(55) 
ip s+r -j(u) 



By the global limit theorem 0, for a€l and j > we get lini ns - 
= %p(u) with ip = ip w from |@SJ|, yielding lim r , s ^oo fj (^s+r-j (^)) = fj(ip(u)) 
Thus, by dominated convergence, for j > 0, 



hm /^(v) = f fj(il>(u))du. 
Using this and the bound (81) there, one can easily verify that 

r OO «7r 

hm V 4% (r, s) = V to^ / (V(u)) du. 



(56) 



(57) 



J'=0 



J=0 
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This gives for Cum in the case a = oo the same formula as written in (|44|l for the 
case 1 < a < oo. Now, instead of l)5()[l . in the Bottcher case we have 

\ip{u)\ < e- Cu \ uER, (58) 

for some constant C, see [El Theorem 23]. Therefore we get again (|49f) and Q52JI 
also in the Bottcher case. Finally, by our Remark below, w(0) = and again we 
arrive at Ci; m = 0. 

It remains to discuss the case a = 1. Here in the last formula at p. 1156 of [To] 
there is a sign error: It must be read as J 7r / m [S(^/'(it)) + S(^>(— w))] du, which equals 
indeed the true value of C\\ m . Now, at least if Z is embeddable into a continuous- 
time Galton- Watson process then analogously to l|14|) we get the identity S(-0(u)) = 
Kq [iu)~ l for some constant K$ > 0, implying Su/>(u)) + S(-0(— u)) = 0. Then 
Ciim = for this class of processes. 

Altogether, all these contradictions to the quoted claim from |15l 'Theorem 1'] 
(and its generalization TjaJ 'Theorem 2']) had been rather unexpected for us. Of 
course, they gave us some more motivation to ask for the right and general picture 
on lower deviation probabilities. Actually, it is wrong to distinguish between veloc- 
ity cases as in (j2EJ. The only needed velocity case differentiation is the mentioned 
dichotomy of Definition^ This we will explain in the next two sections. In the end 
of Section ll~7l we then discuss the influence of ^3 'Theorem 1'] to other results in 

1.6. Lower deviations in the Schroder case. We start by stating our results 
on lower deviation probabilities in the Schroder case. Recall that here — or 1. 

Theorem 4 (Schroder case). Let the offspring law be of the Schroder type and 
of type (d, n). Then for all k n = fi (mode?) with k n — > oo but k n = o(c„), 

nz n = k n ) = — — + ( 59 ) 

m a " c arl \ m n a " c an / v ' 

and 

P(0 <Z n < k n ) = Pfo < W < — ^ ) (1 + o(l)) (60) 

V mr a " Ca n ' 

as n f oo, where for n > 1 fixed we put a n := min{f > 1 : ci > k n }. 

The appearing of the a n in the theorem, depending on the c n and k n looks a bit 
disturbing, so we have to discuss it. First assume additionally that EZilog Z\ < oo. 
Since here we set c n = m™, from Ij59(l we obtain the a„-free formula 

P(Z n = k n ) = dm~ n w(k n /m n ) (l + o(l)). (61) 

Also, comparing this with (J7J, we see that under this Z\ log Z\ -moment condition 
in the Schroder case, m~ n w(k n /m n ) describes not only normal deviation proba- 
bilities but also lower ones. 

On the other hand, without this additional moment condition, recalling property 
l)4b|l. c„ = m n L{m n ) with L slowly varying at infinity. Hence, we have 

1 1 L(m n ) , kn k n L(m n ) 

thus = — — ? '- . 62) 



m n a " c an c n i(rn a '»)' c an m n a " c„ L(m a ") 

Therefore, from l|59|l . 

c n V{Z n = k n ) _ L{m n ) w(k n L(m n )/c n L(m a ")) 



dw(k n /c n ) L(m a ") w{k n /c n ) 



(l + o(l)). (63) 



10 



FLEISCHMANN AND WACHTEL 



Using now (|13|1 . we find 

c n P{Z n = k n ) ( L{m n ) \« V(k n L(m n )/c n L(m a -)) 



dw{k n /c n ) \L(m a ™)J V(k n /c n ) 

Next we want to expel the disturbing a„ from this formula. 

It is well-known (Seneta (1976) ^| p. 23]) that the regularly varying function 
x i ► xL(x) asymptotically equals a (strictly) increasing, continuous, regularly vary- 
ing function x t— > R(x) := xL\(x) with slowly varying L\ . Hence, L{x) ~ L\(x) 
as x t oo. Using now |17l Lemma 1.3], we conclude that the inverse function R* of 
R equals x i— > a:L*(a;), where L* is again a slowly varying function. 

Put x n := R*(k n ). Then k n = x n Li(x n ) by the definition of R*. Recalling that 
x n = k n L*{k n ), we get the identity 

L*{k n )L x {x n ) = 1, n> 1. (65) 

For n fixed, define b n := min {£ > 1 : m e Li(m i ) > £;„}. Combined with x„Li(a; n ) 
= fc„ we get 

TO fc " Li(m fc ") > x n £ x (x n ) > m 6 "- 1 ^^"- 1 ). (66) 

But a; i— > a;Li(a;) is increasing, and the previous chain of inequalities immediately 
gives 

m fc " > x n > m b "- 1 . (67) 

By gg, 

c b +i = ?7i fc " +1 L(m h " +1 ) = m m b "£i(m b ") > fe„ (68) 

Li{m»™) 

for all n sufficiently large. Here, in the last step we used m > 1, that the slowly 
varying functions L and Li are asymptotically equivalent, and the definition of 
b n . Now Cb n +i > k n implies 

b„ + 1 > a n , (69) 
by the definition of a n . On the other hand, 

L 1 (m°» +1 ) = m Ll }™Z + ? c ^ ^ fc « ( 7 °) 
L(m a ™) 

for all n sufficiently large. Here, in the last step we used the definition of a n . This 
gives 

a n + l>b n , (71) 
by the definition of b n . Entering with (|71|l and (|69(l into (|67|l . we get 

> x n > m a,l ~ 2 for all n sufficiently large. (72) 



m 



Therefore, recalling l|65|l . 

L(m a ") ~ L(x„) ~ Li(x n ) ~ 1 as n t oo. (73) 

L (K n ) 

Entering this into (|64() gives 

dw (k n /c n) = W m)£ Wl u(W^ (1 + 0(1)) ' (74) 

which contains L* instead of the a n . 

Note also that such reformulation of l|59|l reminds the classical Cramer theo- 
rem (see, for example, Petrov (1975) §VIII.2]) on large deviations for sums 
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of independent random variables. There the ratio of a tail probability of a sum 
of independent variables and the corresponding normal law expression is consid- 
ered. The crucial role in Cramer's theorem is played by the so-called Cramer series 
•M s ) := SfcLo ^kS k , where the coefficients depend on the cumulants of the 
summands. For the lower deviation probabilities of supercritical Galton- Watson 
processes we have a more complex situation: It is not at all clear, how to find the 
input data L, L*, V [entering into JUJ] based only on the knowledge of the offspring 
generating function /. 

It was already noted after 113( 1 that if Z is cmbeddable into a continuous-time 
Galton- Watson process then V(x) = Vq . Consequently, for embeddable processes, 
(|74|l takes the slightly simpler form 

^Sfeir = [W(M-(i+«u). m 

On the other hand, if V is not constant, the influence of this function on the 
asymptotic behavior of the ratio c n P(Z n = k n )/w(k n /c n ) is relatively small. In- 
deed, from continuity and multiplicativcly periodicity of V(x) we see that < Vi < 
V(x) < V2 < 00, x > 0, for some constants Vi, Vi . Therefore, from ifTIjl . 

| IK )q«]-(i + *)) < ^ (76) 

< Yl[L(m")L'(k n )]"(l + o(l)). 

VI 

Note also that for many offspring distributions the bounds V\ and V% may be 
chosen close to each other. This "near-constancy" phenomenon was studied by 
Dubuc (1982) and by Biggins and Bingham (1991, 1993) HE] 

1.7. Lower deviations in the Bottcher case. Recall that /1 > 2 in the Bottcher 
case. 

Theorem 5 (Bottcher case). Let the offspring law be of the Bottcher type and 
of type (d, fi) . Then there exist positive constants B\ and B2 such that for all 
k n = fi n (modcT) with k n > /i n but k n = o(c n ), 

-B x < liminf M 6 "-"log[c„P(Z n = k n )] (77a) 

nfoo 

< limsu PM b "-"log[c„P(Z„ = k n )} < -B 2 , (77b) 

n't 00 

where b n := min{^ : C£fi n ~ > 2k n }. The inequalities remain true if one replaces 
c n P(Z n = k n ) by P(Z n < kn). 

Let us add at this place the following remark. 

Remark 6 (Behavior of w at 0). In analogy with (|29|l . in the Bottcher case one 
has 

Iog«;(x) x -x-W-fl as i|0 (78) 
with/3 from (gU). This can be shown using techniques from the proof of Theorem[S] 
see Remark HBl below . O 

Our results in the Bottcher case are much weaker than the results in the Schroder 
case: We got only logarithmic bounds. But this is not unexpected, recall our 
discussion around (J2DJ. 
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Repeating arguments as we used to obtain ifFIJl. from Theorem we get 

X °tfc!)- n ^-^ * -{L*(k n /m^)L^H m nY as n Too, (79) 

where L* is such that R\{x) := x^ x ~^ L(x) and R2{x) :— x 1 ^ 1 ^^ L*(x) are asymp- 
totic inverses, i.e. i?i(i?2(^)) ~ x and R2(Ri(x)) ~ x as x | oo. 
Taking into account 1781) . we conclude that 



log[c„P(Z„ = fc„)] 

]ogw(k n /Cn) 



L*(k n /m 0n )L 1 ^ 1 ~^{m n ) as n T oo. (80) 



Let us continue our discussion of the paper JS] . The main reason to study there 
lower deviation probabilities is the application to large deviation probabilities for 
the ratio Z n+ i/Z n , stated as Theorems 3 and 4 there. Using our Theorem 01 
(instead of 'Theorem 1' there) in the proof of |15l Theorem 3] concerning large 
deviation probabilities in the Schroder case, one can easily verify that one needs 
only to change the quantity B in |15l Theorem 3] to be — logpi for all a £ (0, oo), in 
order to get the right picture. On the other hand, ^1 Theorem 4] concerning large 
deviation probabilities in the Bottcher case is true as it is stated, since 'Theorem 1' 
was used only to show that 

lim — log[m"P(Z„ = fc„)l =0 if t > 0, (81) 

n|oo K n K n nfoo 

see ^1 p. 1163]. Recalling that c„ = m n and L(x) = L*[x) = 1 under EZilogZi < 
oo, using our (j79(l . one obtains 

1 /mP n xi/t 1 -/ 3 ) 

— log[m" P(Z„ = kn)] - - ( -t— ) as 7i t oo. (82) 

But mP = [i by definition of /?, and (|8*T|l follows indeed. 



2. Cramer transforms applied to Galton- Watson processes 

Our way to prove Theorems 0] and El is based on the well-known Cramer method 
(see, e.g., ^3 Chapter 8]), which was developed to study large deviations for sums 
of independent random variables. A key in this method is the so-called Cramer 
transform defined as follows. A random variable X(h) is called a Cramer transform 
(with parameter ft G 1) of the random real variable X if 



T? p (h+it)X 

Ve ltx{h) = EehX , teR. (83) 



Of course, this transformation is well-defined if ~Ee hx < oo. 

In what follows, we will always assume that our offspring law additionally sat- 
isfies po = 0. This condition is not crucial but allows a bit simplified exposition of 
auxiliary results formulated in Lemma 1111 below and of the proof of Theorem 01 in 
Section l3~D fsee also Remark ED below). 
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2.1. Basic estimates. Fix an offspring law of type (<i, /i). Let n> 1. Since Z n > 
0, the Cramer transforms Z n (—h/c n ) exist for all h > 0. Clearly, ~E e ttz n{-h/c n ) _ 
fn{e~ h l Cn+lt )l '/n(e _ ' l / c "). We want to derive upper bounds of f n (e~ h / Cn+lt ) on 
{t € M : c^ 1 ^ 1 < \t\ < ttcT 1 } . For this purpose, it is convenient to decompose 
the latter set into Uj=i Jj where 

J 3 {t : c^nd- 1 < \t\ < cj^nd- 1 }, j > 1. (84) 

To prepare for this, we start with the following generalization of |10l Lemma 2]. 

Lemma 7 (Preparation). Fix e £ (0, 1). There exists 9 = 9(e) £ (0, 1) such that 

\U(e- h ' ct+i ^ ct )\ < 6, 1 > 0, h > 0, t G J £ := {t : end- 1 < \t\ < tt^ 1 }. 

Proof. Put g h . t (x) := e - hx+ltx , h,x>0, teR. Evidently, 

\9hAx) ~ 9hAv)\ = \e- hx {e Ux -e lty ) + e^{e~ h * - e~ hy )\ (85) 
< | e *te - e ity \ + \e~ hx - eT hv \ < (h + \t\) \x - y\. 

It means that for if > 1 and T > ttgT 1 fixed, Q := {.g M ; < h < H, \t\ < T} is 

a family of uniformly bounded and equi-continuous functions on R+ . Therefore, 
by ©, 

Me - h/ce+lt /c e) = EflM (Z,/c<) E 5h , t (W0 as £T«), (86) 

uniformly on Q (see, e.g., Feller (1971) [TT1 Corollary in Chapter VIII, §1, p.252]). 
Since W > has an absolutely continuous distribution, and t £ J e implies \t\ < T, 

sup \E e -hW+itWI < L ( 8? ) 
0<h<H, teJe 

From l|86|l and H87|l it follows that there exist Si € (0, 1) and £q such that 

il 

S, 

On the other hand, (Jelo {e~' l/c< ' +it/cf ; ft, > 0, t £ J e } is a subset of a compact 
subset X of the unit disc D, where K does not contain the d th roots of unity. 
Thus for some 82 £ (0, 1), 

sup \f l (e- h l c '- +lt l c ')\ < 62, t<io. (89) 

0<h<H, *GJ e 

In fact, from Definition ^ 

00 

h(z) = ^P(^ = //+jd)z^+^, £>0, z£D, (90) 
j=o 

implying 

00 

\ft(z)\ < \ j2PiZ t = n l +jd)z* i \. (91) 
3=0 

But the latter sum equals 1 if and only if z is a d th root of unity, that is, if it is of 
the form e 2 ™/ d . 

Combining l|88|) and l|89l) gives the claim in the lemma under the addition that 
h < H. Consider now any h > H . In this case 

\f t {e- h l cl+it l ct )\ < Me- 1 /"'). (92) 



sup \Me- h/ce+lt/ce )\ < Si, £>£ . 

0<h<H, t£j. 
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By (J2J we have 

f e (e-' l / ce ) = Ee~' lZ ^ c ' -» Ee-' lW/ G (0,1] as I \ oo, (93) 
uniformly in ft from compact subsets of R + . In particular, 

sup fe{e- 1/ct ) < 1. (94) 

This completes the proof. □ 
The following lemma generalizes [10] Lemma 3]. 

Lemma 8 (Estimates on J\,. . . J n ). There are constants A > and 8 E (0, 1) 

such that for h > 0, t E Jj , and 1 < j < n, 

f Ap?"^ 1 m £/ie Schroder case, 

l /n(e } l ^ • (95) 

I ; in cases. 

Proof. By (l4*ajl . we have e := inf^>i e (0,1)- If £ G Jj , j > 1, then 

evidently, 

ttcT 1 > Cj-i\t\ > Cj-xc^irdT 1 > end' 1 , (96) 
hence Cj-it € J £ . Thus, by Lemma 

OG 

17 := (J |/ J _ 1 (e- /l+it ); ft > 0, £ G J, } C 0D with < < 1. (97) 

From the representation [j9"U|l . /^(z) < for all £ > and |z| < 1. Hence, 

for all z G U C 6>L> we have the bound |/ £ (z)| < 0^'). Thus, for ft > 0, £ € 
J, , and 1 < j < n, 

|/ n (e-^+«)| < / n _ i+1 (e- fc /o»-Ht^ < ^"- +1 ) 7 (98) 

which is the second claim in (|95|l . 

If additionally p\ > 0, then by © (and our assumption po = 0) we have that 
P\ l fi{z) converges as I \ oo, uniformly on each compact K C D°. Therefore, 
there exists a constant C = C(K) such that 

< CpJ, ^> 0, 2 G K. (99) 

Consequently, iterating as in (|98|l . 

|/ n ( e -Vcn+it)| < Cp^ J ' +1 , ft > 0, £ e Jj , 1 < j < n, (100) 

finishing the proof. □ 

2.2. On concentration functions. Fix for the moment ft > and n > 1. Denote 
by {Xj(ft, n)}^ >;L a sequence of independent random variables which equal in law 
the Cramer transform Z n (—h/c n ), that is 

V(X 1 {h 1 n) = k) = _ h/e P(Z n = fc), fc>l. (101) 



Put 



S*(ft,n) := ^X,-(ft,n), £ > 1. (102) 

3=1 
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Note that 

Ee its e (h,n) = (f n ( e - h /^ t )/f n ( e - h / c ")) e . (103) 
Recall notation a £ (0, oo] from (JSJl. 

Lemma 9 (A concentration function estimate). For every h > 0, there is a 
constant A(h) such that 

sup c n P(St(h,n) = k) < ^ , l> £ :=l+[l/a]. (104) 



,fc>i 



Proof. It is known (see, for example, ^3 Lemma III. 3, p. 38]) that for arbitrary 
(real-valued) random variables X and every A,T > 0, 

/Q6\ 2 f T 
Q{X;X) := su P P(y < X < y + X) < (-) max^T" 1 ) J ^ x (t)\dt (105) 

(with the characteristic function of X). Applying this inequality to X = 
Si (h,n) with T — nd^ 1 and A = 1/2, using pt)3|l we have 

su P P(5, (/ l ,n) = fc) < C / ' h/e d * ( 106 ) 

fe>l J-nd- 1 fn {e n/Cn j 

for some constant C independent of h, n. By <|93|l . for ft fixed, /„(e~ /l / c ™) is 
bounded away from zero, and consequently, there is a positive constant C'(h) such 
that 

supP(St (h,n) = k) < C(h) \fn(e- h ^ +it )\ ta dt. (107) 

fc>l J-ird- 1 

Fist assume that a < oo (Schroder case). Using the first inequality in ||SSJ|, we get 
for 1 < J < n, 

\fn(e- h/c " +lt )\ e ° dt < A e ° P { n ~' +1)e °\JA < 2Trd- 1 A e °p[ n -' +1)e °cj\. (108) 
On the other hand, 

7rd _1 /cn 

|/n(e- h/c - +tt )| <G dt < 2nd~ 1 /c n . (109) 
From H108|l and l|109|) . for some constant C, 

| /n(e -Vc„ + «^o di < Cri + ^p^'+^^cTi). (110) 

-7TC* 1 J = 1 

But by l(iajl. 

c n < m n-J ' +1 Cj_i , 1 < j < n. (Ill) 

Also, by the definition of £q in (|104fl and a in (JSJ, pj°m = p}^ 1 ^ 1 ' a < 1. Hence 
the right hand side of (|110fl is bounded in n. Thus, from (|107|) it follows that 

sup c n P(S eo (h,n) = k) < C(h). (112) 

n,k>l 
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This estimate actually holds also in the Bottcher case, where £q = 1. Indeed, 
proceeding in the same way but using the second inequality in 1|95|) instead, the 
sum expression in (|110f> has to be replaced by 

n n n 

J^e^^cncj^ < m n - j+1 = jS^W, (113) 

i=i j'=i j=i 

which again is bounded in n. 

Note that i|112|) is (|104f> restricted to = £q . Hence, from now on we may restrict 
our attention to I > £q . Let Y± , . . . , Yj be independent identically distributed 
random variables. Then by Kesten's inequality (see, e.g., ^3 p. 57], there is a 
constant C such that for < A' < 2A the concentration function inequality 

Q(Y 1 + ... + Y j ;X) < Q(y i; A)[l-Q(y i; A')r 1/2 (114) 

holds. We specialize to Y x = St {h,n) and A' = A = 1/2. Note that Q(Yi; 1/2) = 
sup fe>1 P(Se (h, n) = k) < 1 in this case, since the random variable Xi(h,n) is 
non-degenerate. But also as n | oo this quantity is bounded away from 1, which 
follows from l|112|l . Consequently, inf„>i [1 — Q(Y\\ 1/2)] > 0. Thus, using again 
(|112|) . we infer 



Ci(fe) _ C 2 (h) 

„;^r v ^ ov "' v "' v " j-va ( ^ o) i/2 



sup P^fon) = fc) < = 7 -/^ , j > 1, (115) 



for some positive constants C\{h) and C2(h). If X and Y are independent random 
variables, then, Q(X + Y; A) < Q(X; A) (s. E3 Lemma III.l]). Thus for every 
£ > -^o we have the inequality 

sup c n P(Si(h,n) = k) < sup c„ P(S[e/e„]e (K n) = k) . (116) 

n,fc>l n,fc>l 

Combining this bound once more with (|115l) . the proof is finished. □ 

Remark 10 (Special case h = 0). Note that Se(0,n) equals in law to Z n 
conditioned to Zq — I. Therefore, by Lemma El 

swp-p{Z n =k\Z = t) < ^P-, n>l, £>£ - (117) 



fe>i 



In particular, if a > 1, implying £q = 1, in (|117fl all initial states Zq are possible. 
Especially, if Zq = 1, then inequality l|117|) generalizes the upper estimate in 
(10)] to processes without Z\ log Zi -moment condition. <> 

Lemma can also be used to get very useful bounds for P(Z n — k \ Z$ — £) 
which are not uniform in k. This will be achieved in the next lemma by specializing 
Lemma El to h = 1. 

Lemma 11 (Non-uniform bounds). There exist two positive constants A and S 
such that 

c n P(Z n = k\Z =£) < Ae k/Cn r 1/2 e- se , n,k>l, £>£ , (118) 
[with £q defined in l|104[) ]. 
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Proof. By the branching property and the definition 1102(1 of Si(h,n), 

P(Z n = k\Z =l) = e kh / c -[f n (e^ c ")YP(S e (h,n)=k). (119) 
Putting here h = 1 and multiplying both sides by c„ , we have 

c n P(Z n = k\Z Q =£) < e fc / c "[/„(e- 1 / c ")]'maxc n P(^(l,n) = k). (120) 

Using Lemma El gives 

c n P(Z n = k\Z Q = £) < A(l)t- 1 / 2 e k '°»[f n (e- 1 ' c »)] i . (121) 

From l|94(l the existence of a 5 > follows such that f n {e~ l ' Cn ) < e _<5 for all n > 1. 
Entering this into (|121|l finishes the proof. □ 

2.3. On the limiting density function w. Recall from Section ITTI that w de- 
notes the density function of W, and "0 = V'w its characteristic function. 

Lemma 12 (Bounds for the limiting density). There is a constant A > 
such that 

w* l {x) < A {j w(t)dtj , x>0, ^>£ - (122) 

Proof. Suppose a < oo, the case a = oo can be treated similarly. By the inversion 
formula, 

1 Z" 00 

w* lo {x) = — e- ltx ^ e °(t)dt, x>0. (123) 

Hence, 

i /*°° 

A := supu^ (x) < — / |^(*)| dt. (124) 

x>0 27T J_ 00 

We want to convince ourselves that A < oo. For j > 0, 

\ \ip(t)\°dt = m j / \ij}(tm j )\ dt = m? \ \fj (<i/)(t)) \ ° dt, (125) 

Jmi Jl Jl 

where we used <|47|) . Since W > has an absolute continuous law, \ip(t)\ < C < 1 
for t 6 [ljTir]. Moreover, by l|99|l . < C p[ for z in a compact subset of £>°. 

Therefore, 

/ \^(t)\ io dt < Cm i f{ to = Cm? [1 ~ al ^ (126) 
by definition (jSJ) of a. Consequently, 

/ |V>(*)| dt < C^m^ 1-0 ^ < OO, 



(127) 



since 1 — a£o < 0. Analogously, 

-l 

|V>(t)| dt < oo. (128) 

Hence, A in l|124|l is finite. But w* ( - e+1 \x) = J* w**(x - y)w(y)dy, x > 0, and 
the claim follows by induction. □ 
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2.4. A local central limit theorem. Recall notation Q102(l of Se(h,n), h > 0, 
t,n> 1. By an abuse of notation, denote by ipi — W™ the characteristic function 
of the random variable 

r 1/2 (X~ l {h, n) (S e (h, n) - ES e (h, n)) , (129) 



where a(h,n) := yJE (Xi(h,n) - EXx(h,n)) . Note that by $TU5$ , 

/ , f ( a -h/c rl +ite~ 1/2 cr~ 1 (h.n)\\ 

V /n(e ' l / c ") J 



(130) 



Lemma 13 (An Esseen type Inequality). // < hi < hi < oo, then there 
exist positive constants C — C{h\, h 2 ) and e = e(h\, h 2 ) < 1 such that 

SUp |^'"W-e- t2 / 2 | < Cr 1 /2|t|3 e -* 2 /3 ; \ t \ <E £ 1 / 2 , £> h (131) 

h.e[fei,h. 2 ], n>l 

Proof. Put Xj(h, n) :— Xj(h, n) — ~EXj(h, n). Using the global limit theorem from 
(PJ one easily verifies that for some positive constants Ci, . . . , C4 , 

Ci < < C 2 uniformly in he[h\,h 2 ] and n > 1 (132) 

and 

E|Xi(/i,n)| 3 . 

C3 < — 1 - — < C4 uniformly in h £ [hi,h2\ and n > 1. (133) 

Consequently, the Lyapunov ratio E|Xi(/j, n) | 3 /a 3 (h, n) is bounded away from 
zero and infinity. Applying now Lemma V.l from |16j to the random variables 
X%(h, n), . . . , Xi(h, n) we get the desired result. □ 

The next lemma is a key step in our development concerning the Bottcher case. 
Recall notations Si :— Si(h,n) and a := a(h,n) defined in p()2|l and after (|129|l . 
respectively. 

Lemma 14 (Local central limit theorem). Suppose the offspring law is of type 
(d,/j,). If < hi < h-2 < 00, then 



sup sup 

/ifz [h\ ,/i2] k: k=£[i n (mod t 
n>l 



i 1 ' 2 a(h, n)P(S e (h, n ) = fc) - JL e -x;,,(h,f0/2 

V 27T 



— > 0, 



where x^t := Xf-,e(h, n) := I 1 ' 2 it 1 {h 1 n) (fc — £EXi(/i,n)). 

Note that a local limit theorem, which would correspond to our case ft. = but 
concerning an offspring law with finite variance and with initial state tending to 
00, was derived by Hopfner |121 Theorem 1]. The following proof of our lemma is 
a bit simpler, since for h > the random variables X\(h, n) have finite moments of 
all orders (also if the underlying Z does not have finite variance). 

Proof of Lemma 1141 By (|103fl and the inversion formula, 



1 



/ n ( C -*/c»+«) 



f n (e- h H 
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Decomposing the unit circle, 



d-l 



{e lt : -7T < t < tt} = |J { Q 3 e lt : -ttcT 1 < t < nd' 1 } , (135) 
where g :— e 27T ^ d , the latter integral equals 



3=0 



— jk —itk 

g J e 



—Kd- 1 



f n {e>e- h ' c "+ H ) 



dt. 



(136) 



It is known (see, for instance, ^ p. 105]) that for an offspring law of type (d, /j.) we 
have 

f n (g>z) = f?^7n(z), n,j>l, zeD. (137) 
Therefore the latter sum equals 



7rd 



Trd- 1 



-itk 



h/c n +it\-,l 



f n {e- h ' c -+ lt ) 



fn(e~ h M 



(138) 



3=0 



But Q -3( k -^ n ) ee 1 for k = i[L n (mod d). Altogether, for (Pljl we get 



p(&-») - ^ 



7rd 



TTd- 1 



-itk 



Ue- h ' c -+ u ) 



f n {e- h ' c « 



dt, k = £fj, n (mod d). (139) 



Using the substitution t — > t/l 1 ' 2 a and l|13U|) . we arrive at 



e -«**,/^(f) dt, jfe ee ^™ (mod d). (140) 



Fix < /ii < /i 2 < oo. Recall from ()132|) that 

cr(/i, n) 



d < inf 



< sup 



<j(h, n) 



< C 2 (141) 



he[hi,h 3 ],n>l C n he[hi,h 2 ],n>l c « 

for some < C\ < C 2 (depending on h\, hi). Choose a positive 

£ = e{hi,h 2 ) < Cx-nd- 1 (142) 
as in Lemma IT31 Take any A = A[h\,h,2) > £ (to be specified later). Then the 
identity e _lte_t / 2 dt — \/2n e" x ' "'I 2 and representation (|140|l imply that 



sup 

k: k=£[i n (mod d) 



£ 1/2 aP(S e = k) - 



< d{h+I 2 +h + h), (143) 



where 



h := 



1/2 



ee l / 2 <\t\<Ae 1 / 2 



\Mt)-e- t2 / 2 \dt, I 2 := 



|t|>ef 1 /2 



2 / 2 dt, (144) 



A£ 1 / 2 <\t\<Trd- 1 £ 1 / 2 tr 



\Mt)\dt- 



[Of course, I4 disappears if A(hi,h 2 ) > ird~ 1 a(h,n).} 

Trivially, I 2 — ► as £ | 00. Further, due to Lemma IT3l there is a C = C(h\, h 2 ) 
such that 



Ji < cr 1/2 f 



-1 1 ' 2 



t 3 e -* 2 / 3 dt < cr 1 ' 2 — > 0. 



(145) 
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Thus, it remains to show that the integrals I3 and I4 converge to zero as £ f 00, 
uniformly in the considered ft and n. 

First of all, using again (|130fl and substituting t — » tfi^a/cn, by (|141f> we 
obtain the following estimates 

/n ( e -fc/c-+«/c») 



J 3 < C,^ 1 / 2 



/ B (e-Vc4«/c) 



/e/C 2 <|t|<A/Ci 
"'A/C2<|t|<7rd- 1 c 

First we fix our attention to ^3 . By (|86|) , 

/n(e-' i/c " +rf/c ") -» Ee-' lH/+lW as n t 00 



fn{e~ h M 



dt. 



dt. 



uniformly in ft S [0, ft 2 ] and f e [0, A/Ci] [recall (H2J)]. Also, by ffity. 



/n(e 
It follows that 



/n(e 



Ee 



-h/Cn-\-it/c r . 



-hW 



as n I 00, uniformly in ft G [0, ft 2 ]. 



-£ e -hW+itW 



Ee 



(146a) 
(146b) 

(147) 
(148) 

(149) 



f n (e- h /^) »Too 

uniformly in ft £ [0, ft 2 ] and i€ [0, A/Ci] (with W^— ft) the Cramer transform of 
W). Since the W(—h) have absolutely continuous laws, we have \Ee ltw< --^\ < 1 
for all ft > and |t| > 0. This inequality and continuity of (ft, t) i-> Ee lW( ~' l) 
imply that 

\^ e -hW+itW 

sup 

0<h</i 2 , e/C 2 <|t|<A/Ci 



Ee 



< 1. 



(150) 



Using (|149fl and (|150|) we infer the existence of a positive constant r\ = r\{h\ , h%) < 1 
and an m = ni(fti, ft-2) > 1 such that for n > n\ , 



sup 

0<7i</i 2 , e/C 2 <|t|<A/Ci 



< 



(151) 



/»(e-Vcn) 

Applying (|1 5 If) to the bound of -Z3 in l|146a|) . we conclude that 

h < CAl 1/2 rf -► as ^ too, (152) 

uniformly in ft G [fti, ft 2 ] and n > ni . (The remaining n will be considered below.) 

Next, we prepare for the estimation of J4 . Since / n (e~' l / c ") > iri(e~' i2 / c ") for 
< ft < ft 2 , and f n (e~ h2/Cn ) -> Ee~ ,l2W/ > as n f 00 [recall JT^J], there is a 
positive constant C = C(ft 2 ) such that 

/ n (e- h / c " +rt ) 
f n [e- h ' c -) 

for all t € E, < ft < ft 2 , and n > 1. 

At this point we have to distinguish between Schroder and Bottcher cases. Ac- 
tually, we proceed with the Bottcher case a = 00, which is the only case we need 
later, and leave the other case for the reader. Applying the second case of (19511 to 
(|153|) . we obtain the estimate 

/n(e -v e „+« t) 



< C \f n (e- h ^"+ u )\ 



(153) 



< C exp log (T 1 ], 



(154) 



LOWER DEVIATIONS FOR SUPERCRITICAL GWP 



21 



< h < fi2 , t G Jj , and 1 < j < n. Since /i > 2, there exists an n% — 7^2(^2) 
such that 

fn{e - h /c n+ U } 

if < h < h-2 , t S Jj , and 1 < j < n — n,2 . But | Jj\ < 2cJ 1 1 nd~ 1 . hence 



< expf-zi^ loge*- 1 ]. 



(155) 



f n (e- h/c " +lt ) 



lj 3 fn(e~ h M 
Summing over the considered j gives 

fn(e- h/c - +lt 



dt < 2cj} 1 Trd~ 1 exp[-£ n n ~ j \0g6- 1 ]. (156) 



<rd- 1 <\t\<Trd- 1 



/„(e-Vc») 



dt < 2nd' 1 J2 cJ^expf-^/i^loglT 1 ], 



< h < fi2 and n > 712 . Substituting t — > i/c„ and using 11111) . we arrive at 



/ 

Jnd- 1 



m n 2<|t|<7re2- 1 c„ 
n— ri2 



/n(e -Vc„+ii/c„) 



/«(c- h /o») 



eft 



(157) 



< 2 7 r(T 1 ^ m"- J+1 exp[-^™- J 'log<r 1 ] 



3=1 

OO 



< 2 7 r(T 1 ^m J ' +1 exp[-^V'log<r 1 ] < Ce' ' 1 

3 = 1 

with constants C, C", uniformly in /i G [/ii, ft.2] and n > tt-2 ■ Choosing now A so 
large that ird~ 1 m n ' 2 < A/C2 , we conclude from l|146b|) that 



h < C£ 1/2 e- c ' e ^ as £ T 00, 

uniformly in ft, G [ft-i, /12] and n > ri2 ■ 

Finally, we consider all n < n* := ri\ V 712 . By definition, as in l|9l)|> . 



(158) 



/w ( e -fe/c„+ lt /c n) 

/»(e-*/c) 



^V{X l {h,n) = [j, n + jd) e ( l */ c ")^"+^). (159) 
3=0 

Hence, since the set je~ lt / c " : t G [e/C-z, Trd~ 1 c n ]} does not contain the d th roots 
of unity, 

J n ^ e ~h/c n +it/c„ 



sup 

te[e/C 2 , wd-icn] 



,(/») < 1. 



(160) 



/„(e-"/ c ") 

From the continuity (/i, t) — > f n (e~ fl ' Cn+it ' Cn ) it follows that the function 6*„ is 
continuous, too. Therefore, 



sup # n (/i) 
ft.e[/ii,/i 2 ] 



Combining (|160(l and (|161|l . 



max sup 

te[e/0 2 ,7r(i _1 c„ 



fn{e 



< 1. 



-h/c n -\-it/c n 



fn(e- h M 



< 



for some < 1. Substituting this into (|146fl gives 

h + h < C£ 1/2 6 e as ^ t 00, 



(161) 



(162) 



(163) 
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and the proof is finished. □ 



3. Proof of the main results 

3.1. Schroder case (proof of Theorem[3J). Let /, k n , and a n be as in Theorem^] 

Fix tiq such that c„ > k n > 1 and n > a n > 1 for all n > n , and consider only 
such n. Recall that po = by our convention. By the Markov property, 

oo 

P(Z n = M = E P ( Z «--" = P(^«n = fc « | ^0 = 0- ( 164 ) 

and 

oo 

P(Z n < k n ) = P ( Z n-a n = t) P(^a„ < K \ Z = £) . (165) 



Step 1° (Proof of H59|) ~). Using Lemma ITTI we get for N > £ the estimate 

°° p k n /c an 

c an Y, P ( Z n-a n = I) P(Z an =k n \Z =£) < C — ^ (e- s ) (166) 

for some constant S > 0. By (|4a|) . and since c Qn _i < k n < c a „ by the definition of 



a 



n i 



m -l < < < L (16?) 

On the other hand, by (|9"9"j) . 

/„-a„(e- 5 ) < Cp r r an . (168) 

Thus, from (135)1 . 

oo „ 
K"~"Ca„ 2 P ( Z »-^ = P (^a„ = fc„ | ^0 = ^ < ^ ■ (169) 

By Uni Lemma 9], 

lim j_ r d c " f n{e «/^ )e -it* dt = w ** {x) (170) 

uniformly in x G [wi -1 , 1]. This together with 

C an P(Z an =k n \Z = e) (171) 

= 4~ r /l.(e <t/c ")e-^" /c - dt, I = k n (mod d), 
(see P p. 105]) and (|HT7|) gives 
lim (c a V{Z an = k n \Z Q =t)- dw* £ {k n /c a j) = 0, I = k n (mod d). (172) 

71 too V / 
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Since k n = 1 (mod d), the previous statement holds for all £ = 1 (mod d). For 
other £, the probabilities P(Z n - a „ = £) disappear. Thus, by ((172(1 . 

N-l 

Y,P{Z n _ an =l)P{Z an =k n \Z =e) (173) 
i=i 

N-l 

= dc~l[j2 P(Zn~a n =t)w* e (k n /c an )] (1+0^(1)) 
i=\ 

with o,/v(l)— >0 as n | oo, for each fixed N. Further, using Lemma |12I one can 
easily verify that there exist two constants C and r\ £ (0, 1) such that w* (k n /c an ) < 
Crf for all i> 1 and n. Thus, 

OO OO 

P(Zn-a n = i) W* e (k n / Can ) < P(Zn-a n = I) 4- (174) 

But for every rji € (77, 1), 

f)p(z n _ OB = ^)^ < (±) N U-^m) < c(^yVr a ", (175) 

where in the last step we used ((99(1 . Inequalities il 174(1 and 1(175(1 imply 

00 

J2 P(Zn-a n = I) w*\k n /c an ) < C V T a - e- SN (176) 



e=N 



for all n, N and some constant S > 0. Combining JTHU), (|T75|) . (fH)9"j) and (|T7S|l . 

we have 

00 

P(Z„ = fc„) - dc-^j^P^-^ =f)^(fc n /c j] (l + OA r(l)) (177) 
1=1 

+ 0(c-]p n 1 - a "N- 1 / 2 ), 

where the O-term applies to both n.N f oo. By (|17jl . 

00 

m- 3 w(x/m 3 ) = ^V{Zj =l)w* l {x), j>l, x>0. (178) 

1=1 

Putting here j = n — a n , x = k n /c an , and substituting into 1(177(1 . we arrive at 
P(Z n = k n ) = dc^m"w(k n m a "- n /c arl ) (l + o N (l)) + o(c~^ p^" N' 1 / 2 ) . 
By (fTHTjl . and the definition © of a, 

dc" 1 m a "- n w(fc n r?i a "- r 7c a J > Cc„- 1 m a(a »- n) = C c" 1 pj-"" , for all n. 

(179) 

Therefore, 

P(Z„ = fc n ) = dc-i m a "- n w(k n m a -- n /c an ) (l + o JV (l) + 0(7V- 1 / 2 )), (180) 

where the O-term now applies to N | 00, uniformly in n. Letting first n j 00 and 
then TV | 00, we see that (l5§l) is true. 
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Step 2° (Proof of ijfiTljl). Trivially, for independent and identically distributed non- 
negative random variables X\ , . . . , X n we have 

P(Xi + ... + X n <x) < P(maxXj< x) = P"(Xi < x), x>0. (181) 

3 

Hence, 

P(Z an <k n \Z =£) < P e (Z an < k n ). 
Further, from (|167|l and 10 , 



P{Z an < k n ) < P{c~lZ an < 1) — ► / 

"TOO 



ui(ie) da;. 



(182) 



(183) 



Therefore, since w > on all of (0, oo), there exists an 77 G (0, 1) such that P(Z an < 
k n ) < f] for all n large enough. Thus, 



^ V(Z„-a n = £) P(Z an < k„ \Zq=£) < P ( Z n-a n = Q 

l=N t=N 

for all N sufficiently large. Taking into account (|175J) . we conclude that 

00 

P{Z n - an = I) P(Z an < k n \Z =t) < C V T an e- SN 

for N sufficiently large and some 8 > 0. By the same arguments, 

00 

J2V(Zn-a„=t)F* e (k n /c a J < C P r an e- SN , 



(184) 



(185) 



(186) 



i=N 



where F(x) := P(W < x), x> 0. 

On the other hand, the continuity of F and J2J) yield that P(Z an < c a „x \ Zq = 
£) — > F* l (x) uniformly in x > 0. Therefore, 



lim sup 

"Too fc>! 



P(Z an <k\Z = £)-F* i (k/c an ) 



= 0. 



Combining l(TE5|) . l(T%5|) . (fT5^|) . and (|Tg7f) . we arrive at 

P(Z n < kn) 



(187) 
(188) 



OC 

[^P(Z„_ a „ =£)F* e (kn/c a j] (l + o N (l)) + 0( P r an e- SN ) 



1=1 



with the same meaning of on and the O-term as in the previous step of proof. 
Since P{Z n -a n = 1) = Pi ~ a " and F(k n /c a J > F(m -1 ) > by l(TF7|) . we obtain 



< Ce- SN J2P(Zn-a n =£)F* e (k n /c a J. (189) 

£=1 

Combining this inequality with (|188f) gives 

oo 

P{Z n <k n ) = ^P(Z n -a n =e)F* e (k n /c a j] (l + o N (l) + 0(e~ m )). (190) 

l=\ 

Integrating both parts of (|178J) . one has 

oo 

F(y/m k ) = J2 P ( Z k=Z)F*'(y)> * > L y > 0. (191) 



LOWER DEVIATIONS FOR SUPERCRITICAL GWP 



25 



Thus, 

P(Z n <k n ) = F( h—)(l + ON (l) + 0(e- SN )). (192) 

Letting again first n f oo and then N ] oo finishes the proof. □ 

Remark 15 (Proof in the case po > 0). We indicate now how to proceed with 
the proof of Theorem 0] in the remaining case pa > 0. Here in the representation 
(|164|) one has additionally to take into account that 

P(Z an = k n \Z Q = £) (193) 

= E(0 / ^ (o H 1 -/-(°)) ip {E^ = fe ™|< )>0 ' !<*<i}. 

3 = 1 i=l 

where the Z^ 2 \ . . . are independent copies of Z. Then instead of Lemma ITT1 
we need 

j 

c n P{ Z fl = k n | Zl2 > 0, 1 < i < J } < A e^^r 1 ' 2 e- se , n, k > 1, j > to ■ 

i=l 

But this is valid by 

1 — Jn(y) "Too 1—? 

uniformly in z from compact subsets of D°. This indeed follows from l|9|l. O 

3.2. Bottcher case (proof of Theorem [SJ). From the Markov property, 

00 

P(Z n = k n ) = P(Z n -b n =e)P{Z bn =k\Z =l). (195) 

t=Hn-b n 

Using i|119|) and Lemma |5J we obtain the following estimate 

c b P{Z K =k n \Z = £) < A(h)r 1 l 2 [e hk «l^f K {e- h l c ^)} 1 . (196) 
From the definition of b n it immediately follows that 

2fc„ < c Kl x n ~ h - = Cbn _ 1 ^-bn+if_2h_J) < 2k n -. (197) 

Hence, 

£ * \ < 198 > 

for £ > ^i"" 6 " . Therefore, 

c fc „P(Z fc „ = kn \Z = t) < A{h)r 1 t 2 [e h / 2 f bn {e- h/cb ")} 1 . (199) 

It is known (see, for example, pQ, Corollary III. 5. 7), that EVF = 1 if Epilog Z\ < 00 
and EW — 00 otherwise. It means, that for the Laplace function (p = ipw of W 
we have e h / 2 tp(h) < 1 for all small enough h. Thus, due to the global limit theorem 
©, there exist 6 < 1 and h > such that e h °' 2 f n (e~ ho/c ~) < e~ & for all large 
enough n. Hence, 

c K P{Z K = k n \Z Q = 1) < Ar^ 2 e- Se . (200) 
Inserting l|200|l into (|195|) . we obtain 

c b P{Z n = k n ) < v4 M -( n -"")/ 2 /n- 6 „(e- 5 ), (201) 
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consequently, 

^- n \og[c n P(Z n = k n )} < /A-C + M^logf^) + l0gf ;X S) ■ (202) 



Since c n /c bn < m n - b ™ and ^"~ b " = m^ n - b "\ /A" n log(c n /c 6 J -> as n | oo. 
Thus, 

limsup/"-"log[c n P(Z n = jfc„)] < limsup l0g ^";^ e ^ . (203) 

nToo nfoo p 71 

Using l|21(l . we arrive at the desired upper bound. 

We show now that l|77b|) holds for logP(Z„ < k n ). First of all we note that for 
arbitrary non-negative random variable X and all x, h > 

P(X < x) < e hx Ee- hX . (204) 

Applying this bound to the process Z starting from £ individuals and taking into 
account (|198[1 . we have 

P(Z K <k n \Z =t) < [e hk ^nf bn ( e -Vc b n)Y < [e h / 2 f bn (e~ h / c ^)] e . (205) 

As we argued in the derivation of l|200l) . this gives 

P{Z bn < k n \Z =£) < e~ M . (206) 

Consequently, by the Markov property, 

P(Z n <k n ) < fn-b n {e~ S )- (207) 

Taking logarithm and using 1|21|) . we obtain (|77b(l . 

Let us verify the lower bounds in Theorem [5] By H195|) . 

P(Z n = k n ) > P{Z n _ bn = n n ~ K )P(Z bn = k n I Z = M "- b "). (208) 

From UIH), 

P(Z K = k n | Z = /i"-"") > [f bn (e~ h/Cbn )Y" P(S*„(M») = k n ), (209) 

where l n — /i n ~ h ™ . 

Consider the equation 



C^EXtfabn) = ^ , , ' : - X. (210) 

/' ( e - h / c ^)e- h / Cb 



Evidently, 



and 



hAe- h/cb ") 
f' b { e - h ' c ^)e- h / c ^ 



= m b " (211) 

h=0 



H b '\ (212) 



From these identities and monotonicity of f b (e~ h l Cb ^ ) e -V c i>n / f b ^ (e~ h ' C6 « ) it fol- 
lows that H210fl has a unique solution h n (x) for /i b "c b _1 < a; < m brl c b ~ 1 . Analogously 
one shows that the equation (p'(h)/ip(h) — —x has also a unique solution h(x). By 
the integral limit theorem (J3J, the right-hand side in 12101) converges to —<p'(h) /f{h) 
and consequently, h n (x) — > h{x) as n t oo. Further, by (flUTI) . 

£ s * = ^ 5 I (213) 



LOWER DEVIATIONS FOR SUPERCRITICAL GWP 



27 



Thus, 

%/2m) < liminf fo„ < liminf /i„ < fc(l/2), (214) 

nfoo n|oo 

where /i„ := h n (x n ). It means that there exist /i* and /i* such that h* < /i„ < h* 
for all n > 1. From the definition of h n and (|2 10|> immediately follows that 
ES f £ ri (/i„, 6„) = fc„ . Thus, applying Lemma ITU we get 

£ 1 J 2 a(h n ,b n )P(S en {h n ,b n ) = kn) - 



lim 

ntoo 



= 0. (215) 



/2tt 

Recall that by Q132II we have cr(h n ,b n ) > C c bn . Hence, 

limiirf£l/ 2 c bn P(S in (h n ,b n ) = k n ) >C > 0. (216) 

nfoo 

Moreover, since fb n {e~ hn " /c S > fb n ( e ~ h * /c " n ) and fj{e- h */ c ') E e - h * w > 0, 
there exists a 6* > such that 

/b„(e-' l / c ^) > (217) 
for all n. Applying these bounds to the right-hand side in (|2t)9(l . we find that 

liminf /A'" log [c„ P(Z K = k n | Z = fi n - K )] > -C. (218) 

n'foo 

Using this inequality and l|21|l to bound the right-hand side in 1208JI . we conclude 
that 

liminf/"~"log[c n P(Z„ = kn)] > -C, (219) 

nfoo 

i.e. I|77a|l is proved. 

Next we want to extend this result to P(Z n < k n ). Obviously, 

P(Z n < K) > P(Zn- bn = in) P(Z bn <k n \Z = £„). (220) 

Then, using i|119|) with h = h n , we have 

P(Z n < K) > P(Z n _ bn = In) [f n (e- h ^ c ^)Y" P(%(/l,6 n ) < fc„) . (221) 
By the central limit theorem, 

limP(S,„(Mn) < K) = \ . (222) 

n'foo Z 

From this statement and (|217|l it follows that 
liminf /"""log P{Z n < k n ) > liminf /i 6 """ log P(Z n - bn = ^ bn )+log9. (223) 

n|oo nToo 

Recalling l|17|) . the proof of Theorem is complete. □ 

Remark 16 (To the proof of Remark^. To prove (|78(l one can use the methods 
from the proof of Theorem But some changes are needed, since in Remark [H] we 
deal with absolutely continuous distributions. 

Instead of 1|195|) we shall use l|178|) . Putting there x ~ ym k and k — k y — 

max{j > 1 : m3 < fi> J /2y} we obtain 

oo 

w (y) = m k y P(Z ky =£)w* e (ym k y). (224) 
l=fj, k y 

For every h > we may define the density function 

w h (x) ~ w(x), (225) 



28 



FLEISCHMANN AND WACHTEL 



corresponding to the Cramer transform of W. Bv Lemma ll2l C w :— sup x>0 w(x) < 
oo in the present Bottcher case. Hence, sxvp x>0 Wh(x) < C w /<p(h). By induction 
(analogously to Lemma |5J), 

supwfix) < 1> 1. (226) 

x>0 <P(h) 



< l W = ~~T7TX l > 1 - ( 227 ) 

<r{h) 



-ha. 



It is easy to see that 



From this identity and (|226[1 it follows that 

w*\x) < C w ^- l {h)e hx , (228) 

Therefore, for all I > fi ky , 

p hyir h " 

<p(h) 

Further, by the definition of k y , 



w* e (ym ky ) < -% \e hvmky ^ hy ^(h)]\ (229) 



a m y 1 

< -T- < TT . (230) 



2my [i ky 2y ' 
and consequently, 

w*\ym ky ) < ^- [e h /Mh)Y- (231) 
Before <|2(J(J[) we showed that e h °/ 2 (p(ho) < e~ s . As a result we have the bound 

w* e (ym ky ) < 4rr (232) 
if [ho) 

Entering this into (|224|l gives 

w(y) < Cm k y f ky (e- s ). (233) 
Taking logarithm and using 1211) . we see that 

limsup iT k * logw(y) < logB(e"' 5 ). (234) 

Now we deal with a corresponding lower bound of log w(y). By l|224|l and (|227|l . 
w(y) > m fc » P{Z ky = ii ky )w*^ kv {ym ky ) 

> P(Z ky = fi ky )^ ky (h) w*/ v (ym ky ), h> 0. (235) 

Recalling that h{x) is the unique solution of the equation tp'(h)/ip(h) = —x and us- 
ing i|230|) . one gets the inequality h(ym ky / fi ") < h(fi/2m). Thus, by monotonicity 
of <p, 

ip^ ky (h(ym ky /n ky )) > <p^ ky {h{n/2m)) = exp[-C/i fe "]. (236) 

If in l|225|l we set h — h(ym ky //i fc "), then wV 1 v (ym ky ) is the value of the density 

function of the sum J2j=i h) at the point E^^'J^ Wj(— h). Thus, by the 
central limit theorem for densities f |16l Theorem VII. 7]), 

\imwf y (ym ky ) = -±= . (237) 
v2tt 
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Putting h = h(ym k y /pi k y) in (|^5| and using (UJ, (21J, and it^Tfjl . we obtain 

liminf fj.- k y logw{y) > -C. (238) 
v->o 

Combining (|234[> and (|238[1 we get 

\ogw(y) x -//*• (239) 
Then the relation [j, ky x y-P/^-P) finishes the proof. O 
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